Skip to content

docs: add release notes and docs for batched shuffle insertion (PR #1369) - #1698

Merged
lbliii merged 1 commit into
26.04-stagingfrom
lbliii/docs-rio-classifier
Apr 3, 2026
Merged

docs: add release notes and docs for batched shuffle insertion (PR #1369)#1698
lbliii merged 1 commit into
26.04-stagingfrom
lbliii/docs-rio-classifier

Conversation

@lbliii

@lbliii lbliii commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Description

Adds 26.04 release notes and documentation for the batched shuffle insertion feature introduced in PR #1369. The release notes describe the new identification_batchsize parameter and batch-aware shuffle adapter. The exact dedup configuration table and performance best practices are updated accordingly.

Usage

exact_workflow = ExactDeduplicationWorkflow(
    input_path="input_data/",
    output_path="./results",
    text_field="text",
    input_blocksize="256MiB",
    identification_batchsize=4,  # Process ~1 GB per insertion call
    input_filetype="parquet"
)

Checklist

  • I am familiar with the Contributing Guide.
  • New or Existing tests cover these changes.
  • The documentation is up to date with these changes.

@lbliii
lbliii requested a review from a team as a code owner March 31, 2026 19:15
@lbliii
lbliii requested review from meatybobby and removed request for a team March 31, 2026 19:15
@greptile-apps

greptile-apps Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds release notes and documentation for the batched shuffle insertion feature (identification_batchsize) introduced in PR #1369 for ExactDeduplicationWorkflow. The changes are purely documentation — two MDX files are updated to describe the new parameter, its performance trade-offs, and revised best practices for block sizing.

  • Adds a new ## Improvements section to the v26.04 release notes describing the identification_batchsize parameter and the batch-aware shuffle adapter fallback behavior.
  • Adds identification_batchsize to the configuration reference table in exact.mdx with type, default, and description.
  • Updates the performance best-practices bullet in exact.mdx to recommend smaller input_blocksize values paired with a larger identification_batchsize (targeting 2–6 GB per insertion call) instead of the previous 2GiB default recommendation.
  • The two identification_batchsize examples used within exact.mdx (value 4 in the table, value 8 in best practices) are mathematically consistent but point readers at different starting points, which could create confusion.

Confidence Score: 5/5

Documentation-only PR with no code changes; safe to merge.

All findings are P2 style/consistency suggestions. There are no logic errors, broken links, incorrect parameter descriptions, or code changes of any kind. The documentation accurately describes the feature and its trade-offs.

No files require special attention; minor example-value alignment between the config table and best-practices bullet in exact.mdx would improve clarity.

Important Files Changed

Filename Overview
fern/versions/v26.04/pages/about/release-notes/index.mdx Adds a new "## Improvements" top-level section for batched shuffle insertion; content is accurate and consistent with the feature PR, but the section sits at the same heading level as "What's New" while a similar improvement (Actor Pool Progress Bars) is nested inside "What's New" as a ### subsection.
fern/versions/v26.04/pages/curate-text/process-data/deduplication/exact.mdx Adds identification_batchsize to the configuration table and updates the performance best-practices bullet; the two examples in the same file use different batchsize values (4 vs 8) which could confuse readers even though both are mathematically correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[ExactDeduplicationWorkflow] -->|reads blocks of input_blocksize| B[Input Blocks]
    B -->|batched by identification_batchsize| C{Batch Ready?}
    C -->|No — accumulate more blocks| B
    C -->|Yes — batch full| D[Concatenate Blocks into Batch]
    D --> E{Stage has read_and_insert_batch?}
    E -->|Yes| F[read_and_insert_batch\nbatch-aware path]
    E -->|No — fallback| G[Single-task insertion]
    F --> H[Shuffler / GPU Operations]
    G --> H
    H --> I[ExactDuplicateIds output]
Loading

Reviews (2): Last reviewed commit: "docs: add release notes and docs for bat..." | Re-trigger Greptile

- **Xenna-managed CUDA devices**: Xenna now manages CUDA device visibility directly, replacing the previous Ray-managed approach.
- **Ray 2.54**: Updated Ray dependency to version 2.54 for compatibility with Cosmos-Xenna 0.2.0.

### Batched Shuffle Insertion for Exact Deduplication (PR #1369)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think of this more as an enhancement rather than what's new

Comment thread fern/versions/v26.04/pages/curate-text/process-data/deduplication/exact.mdx Outdated
)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants